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We prove that mutual information is actually negative copula entropy, based on which a method 
for mutual information estimation is proposed. 

PACS numbers: 05.90.+m, 02.50.-r, 87.10.-e 
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I. INTRODUCTION 

In information theory, mutual information (MI) is a 
difference concept with entropy. In this paper, we prove 
with copula that they are essentially same - mutual 
information is also a kind of entropy, called copula en- 
tropy. Based on this insightful result. We propose a sim- 
ple method for estimating mutual information. 

Copula is a theory on dependence and measurement of 
association. 0| Sklar ^ proved that joint distribution D 
can be represented with copula C and margins F in the 
following form: 

Z?(x) -C7(Fi(a;i),...,FA,(xjv)). 

Derived by separating the margins from joint distribu- 
tion, copula has all the dependence information of ran- 
dom variables, which is believed that mutual information 
does as well. 

Here gives notation. C, c denote copula function 
and copula density; D, F denotes joint distribution and 
marginal distribution; H, I, He denote entropy, mutual 
information, and copula entropy respectively. Finally, 
bold letters represent vectors while normal letters single 
variable. 



II. THEOREM AND PROOF 

For clarity, we give directly the main results. Please 
refer to and references therein for more about copula. 

Definition 1 (Copula entropy). Let X be random vari- 
ables with marginal function u and copula density c(u) . 
Copula entropy of X is 



Proof. 



ffc(x) 



c(u) logc(u)du. 



(1) 



Tiieorem 1. Mutual Information of random variables 
equals to the negative entropy of their corresponding cop- 
ula function: 



/(x) 



i?c(x). 



(2) 



-^W = / p(x) log 



p(x) 



-c?x 



) J]^ Pi (a;^ ) log c(u^ ) dx 

i 

c{urc)logc{ux)du^ 

= -i/c(x) 



□ 



Entropy is the information contained in joint density 
function and marginal densities, while copula entropy is 
the information contained in copula density. The follow- 
ing corollary show the relation between them. 



Corollary 1. 



H(x) 



Hix,) + fl-e(x) 



(3) 



Proof. The corollary is an instant result from the defini- 
tion of mutual information and theorem [1] □ 

The worthy-a-thousand-words results cast insight into 
the inner relation between mutual information and cop- 
ula, and hence builds a connection between information 
theory and copula theory. 



III. ESTIMATING MUTUAL INFORMATION 
VIA COPULA 

With theorem[Tl we propose the methods of estimating 
mutual information from data. The estimation composes 
of two steps: 

1. estimating empirical copula density; 

2. estimating copula entropy. 

Given samples {xi, . . . , x^} i.i.d. generated from X — 
[xi, . . . , xn], we can easily derive empirical copula den- 
sity using empirical functions 
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1 ^ 

F,{x,) = -J2x{Xl<xl) 



(4) 
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Mutual information estimation on Gaussian distribution 
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FIG. 1: Mutual information estimation. Black represents an- 
alytic value, red represents by knn, and blue represents by 
copula entropy. 



where i = 1,...,N, and x is indicator function. Let 
u = [Fi, . . . , Fjv] , and then we can derive a group of 
samples {ui, . . . , ut} from empirical copula density c(u). 

Since entropy estimation is a much contributed topic, 
copula entropy can be achieved by well-established meth- 
ods. In the following experiment, we adopt the method 
proposed by 

IV. EXPERIMENTS 

To evaluate the new estimation method, we consider 
tow correlated standard Gaussian variables with covari- 
ance p, of which mutual information is — ilog(l — p^)- 
In the experiments p is set from to 0.9 with step 0.1. 
With each value, we generate a 1000 samples set. Copula 
entropy is estimated by the method in Q. As a contrast, 
mutual information estimation in [3| was also run on the 
same sample set. Figure ITVl illustrates the experimental 
results. It can be learned that our method provides a 
competitive way of MI estimation. 
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